⚡️ Speed up function all_columns_match by 147%
#43
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The optimized code achieves a **146% speedup** through two key optimizations that eliminate redundant computations: **1. Optimized `unq_columns` function:** - **Original**: Created two `OrderedSet` objects and used set subtraction: `OrderedSet(col1) - OrderedSet(col2)` - **Optimized**: Creates only one `set(col2)` and uses list comprehension with membership testing: `OrderedSet(c for c in col1 if c not in col2_set)` - **Why faster**: Set membership testing (`c not in col2_set`) is O(1) on average vs. the overhead of creating multiple OrderedSet objects and performing set arithmetic **2. Completely reimplemented `all_columns_match` function:** - **Original**: Called `unq_columns()` twice, effectively calling `fa.get_column_names()` four times total and performing complex OrderedSet operations - **Optimized**: Calls `fa.get_column_names()` only twice (once per dataframe) and directly compares `set(col1) == set(col2)` - **Why faster**: The line profiler shows `fa.get_column_names()` is expensive (~10ms per call). Reducing from 4 calls to 2 calls plus using simple set equality eliminates the computational overhead of OrderedSet operations entirely. **Performance impact**: The profiler data shows the original `all_columns_match` spent 100% of its time calling `unq_columns`, which in turn spent 99.8% of its time in `fa.get_column_names()`. The optimized version eliminates half of these expensive calls and replaces complex OrderedSet arithmetic with fast set operations. This optimization is particularly beneficial for workloads that frequently check column matching between dataframes, as it reduces both the computational complexity and the number of expensive external API calls.
codeflash-ai
bot
added
⚡️ codeflash
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📄 147% (1.47x) speedup for
all_columns_matchindatacompy/fugue.py⏱️ Runtime :
1.22 milliseconds→496 microseconds(best of5runs)📝 Explanation and details
The optimized code achieves a 146% speedup through two key optimizations that eliminate redundant computations:
1. Optimized
unq_columnsfunction:OrderedSetobjects and used set subtraction:OrderedSet(col1) - OrderedSet(col2)set(col2)and uses list comprehension with membership testing:OrderedSet(c for c in col1 if c not in col2_set)c not in col2_set) is O(1) on average vs. the overhead of creating multiple OrderedSet objects and performing set arithmetic2. Completely reimplemented
all_columns_matchfunction:unq_columns()twice, effectively callingfa.get_column_names()four times total and performing complex OrderedSet operationsfa.get_column_names()only twice (once per dataframe) and directly comparesset(col1) == set(col2)fa.get_column_names()is expensive (~10ms per call). Reducing from 4 calls to 2 calls plus using simple set equality eliminates the computational overhead of OrderedSet operations entirely.Performance impact: The profiler data shows the original
all_columns_matchspent 100% of its time callingunq_columns, which in turn spent 99.8% of its time infa.get_column_names(). The optimized version eliminates half of these expensive calls and replaces complex OrderedSet arithmetic with fast set operations.This optimization is particularly beneficial for workloads that frequently check column matching between dataframes, as it reduces both the computational complexity and the number of expensive external API calls.
✅ Correctness verification report:
⚙️ Existing Unit Tests and Runtime
test_fugue/test_duckdb.py::test_all_columns_match_duckdbtest_fugue/test_fugue_pandas.py::test_all_columns_match_nativetest_fugue/test_fugue_polars.py::test_all_columns_match_polarstest_fugue/test_fugue_spark.py::test_all_columns_match_spark⏪ Replay Tests and Runtime
test_pytest_teststest_sparktest_helper_py_teststest_fuguetest_fugue_polars_py_teststest_fuguetest_fugue_p__replay_test_0.py::test_datacompy_fugue_all_columns_matchTo edit these changes
git checkout codeflash/optimize-all_columns_match-mi6hq9u0and push.